Crowdstrike.fdr: map CommandHistory events to ECS process fields - #20285
Crowdstrike.fdr: map CommandHistory events to ECS process fields#20285chemamartinez wants to merge 9 commits into
Conversation
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
| - script: | ||
| tag: script_process_interactive_logon_type_9b3d6c7a | ||
| description: Set process.interactive from crowdstrike.LogonType; types 2 (Interactive), 10 (RemoteInteractive), and 11 (CachedInteractive) indicate an interactive shell session. | ||
| if: ctx.crowdstrike?.LogonType instanceof String && ctx.crowdstrike.LogonType != '' |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml:1148
The process.interactive script fires on any event carrying a LogonType, so authentication events (UserLogon/UserLogoff) get a bare process.interactive with no process context; gate it on an established process instead.
Details
The new script's condition is only ctx.crowdstrike?.LogonType instanceof String && ctx.crowdstrike.LogonType != '', which is not scoped to process events. LogonType is present on Windows authentication events, so this stamps process.interactive onto documents that have no process. This is observable in the regenerated fixtures: in test-fdr.log-expected.json a UserLogoff event (event.category: ["authentication"]) now gains a lone process: { "interactive": false } object, and test-user-map / test-windows show the same on non-process events. The effect is also broader than the changelog entry ("Map CommandHistory events to ECS process fields") describes. Attaching a process attribute to an authentication event is semantically misleading and creates a process object where none belongs.
Recommendation:
Gate the script on an already-established process so it only applies to genuine process events (CommandHistory and ProcessRollup set process.entity_id from TargetProcessId earlier in the pipeline; authentication events do not):
- script:
tag: script_process_interactive_logon_type_9b3d6c7a
description: Set process.interactive from crowdstrike.LogonType; types 2 (Interactive), 10 (RemoteInteractive), and 11 (CachedInteractive) indicate an interactive shell session.
if: >-
ctx.crowdstrike?.LogonType instanceof String && ctx.crowdstrike.LogonType != '' &&
ctx.process?.entity_id != null
source: |-
def logonType = ctx.crowdstrike.LogonType;
ctx.process.interactive = (logonType == '2' || logonType == '10' || logonType == '11');
on_failure:
- append:
field: error.message
value: "Failed to set process.interactive from LogonType: {{{_ingest.on_failure_message}}}"🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
falcon |
5102.04 | 3378.38 | -1723.66 (-33.78%) | 💔 |
To see the full report comment with /test benchmark fullreport
| if (ctx.process == null) { | ||
| ctx.process = new HashMap(); | ||
| } |
There was a problem hiding this comment.
| if (ctx.process == null) { | |
| ctx.process = new HashMap(); | |
| } | |
| ctx.process = ctx.process ?: [:]; |
| source: |- | ||
| def logonType = ctx.crowdstrike.LogonType; | ||
| if (ctx.process == null) ctx.process = new HashMap(); | ||
| ctx.process.interactive = (logonType == '2' || logonType == '10' || logonType == '11'); |
There was a problem hiding this comment.
I think this wants a comment explaining the logon types.
| // 7=Unlock, 8=NetworkCleartext, 9=NewCredentials, 10=RemoteInteractive, | ||
| // 11=CachedInteractive, 12=CachedRemoteInteractive, 13=CachedUnlock | ||
| def logonType = ctx.crowdstrike.LogonType; | ||
| ctx.process.interactive = (logonType == '2' || logonType == '10' || logonType == '11' || logonType == '12'); |
There was a problem hiding this comment.
Severity: 🟠 High confidence: medium path: packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml:3258
process.interactive is derived from crowdstrike.LogonType on every event that has any process object, which labels processes that are not the session's shell — restrict the processor to CommandHistory events.
Details
LogonType is an attribute of the Windows logon session, but ECS defines process.interactive per-process ("inferred from the processes file descriptors... the character device for the controlling tty is the same as stdin and stderr") — the definition this PR adds to docs/README.md line 3197. Applying the session's LogonType to whatever process object happens to be on the event produces wrong values, and the PR's own fixtures show it:
- packages/crowdstrike/data_stream/fdr/_dev/test/pipeline/test-windows.log-expected.json line 11952 now asserts "interactive": true for the Windows System process (event_simpleName LogonBehaviorCompositionDetectInfo, ImageFileName/CommandLine "System", entity_id 21475403566). System is the kernel process; it has no controlling tty and can never be attached to an interactive shell. It is marked interactive purely because the session's LogonType is 2.
- test-fdr.log-expected.json lines 8069, 8557, 9370 and test-windows.log-expected.json lines 8756, 8963, 9153 add "interactive": false to UserLogon/authentication events whose process object is only a ContextProcessId reference (entity_id plus thread.id). ECS false means "known to be non-interactive"; here interactivity is simply unknown, so the pipeline is asserting a fact the source event does not carry.
The ctx.process != null gate added in this revision keeps the field off events with no process object at all, but it does not distinguish the process the logon session actually belongs to from an unrelated context/target process reference, so both cases above still fire.
Recommendation:
Scope the processor to the event type this PR is about (CommandHistory), where the recorded shell genuinely is the process the logon session belongs to:
- script:
tag: script_process_interactive_logon_type_9b3d6c7a
description: >-
Set process.interactive from crowdstrike.LogonType for CommandHistory
events only. On other FDR events LogonType describes the logon session,
not the process referenced by ContextProcessId/TargetProcessId, so the
value would not describe that process. Logon types 2 (Interactive),
10 (RemoteInteractive), 11 (CachedInteractive) and 12
(CachedRemoteInteractive) indicate an interactive shell session.
if: >-
ctx.crowdstrike?.event_simpleName == 'CommandHistory' &&
ctx.crowdstrike?.LogonType instanceof String &&
ctx.crowdstrike.LogonType != '' &&
ctx.process != null
source: |-
def logonType = ctx.crowdstrike.LogonType;
ctx.process.interactive = (logonType == '2' || logonType == '10' || logonType == '11' || logonType == '12');
on_failure:
- append:
field: error.message
value: "Failed to set process.interactive from LogonType: {{{_ingest.on_failure_message}}}"This reverts the added "interactive" lines in test-fdr.log-expected.json and test-windows.log-expected.json and keeps only the two CommandHistory assertions in test-fdr-command-history.log-expected.json.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
There was a problem hiding this comment.
That's intentional. We want to map process.interactive in every event it contains crowdstrike.LogonType and contains a process object.
| if (!remaining.trim().isEmpty()) { | ||
| commands.add(remaining.trim()); | ||
| } | ||
| ctx.process.command_line = commands.size() > 0 ? commands.get(0) : history.trim(); |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml:1083
process.command_line is set to commands.get(0), an arbitrary entry of a multi-command history buffer, silently dropping the rest — only populate it when the history holds a single command.
Details
For the two-command fixture in test-fdr-command-history.log, process.command_line becomes "sc stop CSFalconService" and the second command, "Stop-Service -Name CSFalconService -Force", never reaches process.command_line (test-fdr-command-history.log-expected.json line 102). Detection rules and searches on process.command_line therefore see only one of the N commands, chosen by position.
Index 0 is not a defensible choice: the same event carries FirstCommand, LastAdded, LastDisplayed, CommandSequence and CommandCount precisely because the history is an indexed rolling buffer, so element 0 is not reliably the newest, the oldest, or the command of interest. The previous revision assigned the whole delimited string; this revision narrowed it to element 0, which trades one wrong value for another.
ECS defines process.command_line as "Full command line that started the process". The process on these events is identified by TargetProcessId (process.entity_id 7292754410393 in the fixture) — the shell — and its own command line is none of the history entries. A related symptom: this script is placed at line 1054, after split_command_line_c3beef26 (line 972), so unlike every other source of process.command_line in this pipeline the value produces no process.args/process.args_count; the fixture's process object has command_line but no args.
The full parsed history is already preserved at crowdstrike.CommandHistory, so nothing is lost by not guessing.
Recommendation:
Keep the parsed array and only expose process.command_line when it is unambiguous:
source: |-
def history = ctx.crowdstrike.CommandHistory;
def commands = new ArrayList();
def remaining = history;
int idx = remaining.indexOf("\u00b6");
while (idx >= 0) {
def cmd = remaining.substring(0, idx).trim();
if (!cmd.isEmpty()) {
commands.add(cmd);
}
remaining = remaining.substring(idx + 1);
idx = remaining.indexOf("\u00b6");
}
if (!remaining.trim().isEmpty()) {
commands.add(remaining.trim());
}
ctx.crowdstrike.CommandHistory = commands;
// A history with several entries has no single command line; picking an
// element by position would drop the others from process.command_line.
if (commands.size() == 1) {
ctx.process = ctx.process ?: [:];
ctx.process.command_line = commands.get(0);
}If you do keep populating process.command_line, move this script above rename_crowdstrike_CommandLine_to_process_command_line_307047e3 (line 967) so split_command_line_c3beef26 also derives process.args and process.args_count, as it does for every other command line in this pipeline.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| - name: text | ||
| type: match_only_text | ||
| - name: CommandSequence | ||
| type: match_only_text |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: medium path: packages/crowdstrike/data_stream/fdr/fields/fields.yml:233
crowdstrike.CommandSequence is declared as match_only_text, so the sequence index cannot be filtered exactly, sorted or aggregated — declare it as keyword.
Details
CommandSequence carries the position of the entry within the CrowdStrike command-history buffer ("1" in both events of test-fdr-command-history.log). match_only_text is a text type: it is not doc_values-backed, so the field cannot be used in a terms aggregation, cannot be sorted on, and cannot be matched with an exact term query — all of which are the natural uses of a sequence index when reconstructing command order.
The adjacent CommandCount and CommandCountMax entries (fields.yml lines 221-226) are also match_only_text, but each sits above a commented-out type: long declaration, which marks them as a known legacy compromise rather than the convention to follow for a newly added field.
Recommendation:
Declare the sequence index as a keyword:
- name: CommandSequence
type: keywordThe docs/README.md row for crowdstrike.CommandSequence needs regenerating with elastic-package build after the change.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
|
✅ All changelog entries have the correct PR link. |
| } | ||
| ctx.crowdstrike.CommandHistory = commands; | ||
| ctx.process = ctx.process ?: [:]; | ||
| ctx.process.command_line = commands; |
There was a problem hiding this comment.
Severity: 🟡 Medium confidence: medium path: packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml:1083
process.command_line is now assigned the whole command array, so the field is multi-valued on CommandHistory events but a scalar string everywhere else in the data stream; populate it only when the history holds exactly one command.
Details
ECS defines process.command_line as the full command line that started the process identified by process.entity_id. On a CommandHistory event process.entity_id is TargetProcessId (the shell), and crowdstrike.CommandHistory is the buffer of commands typed inside that shell, not the command line that launched it. Assigning the array puts N unrelated command strings into a single-valued ECS field.
This is visible in the committed fixtures: test-fdr-command-history.log-expected.json now carries process.command_line as a two-element array, and test-windows.log-expected.json turns a previously scalar value into a one-element array, while every other event in the same expected file (for example the LogonBehaviorCompositionDetectInfo document with "command_line": "System") keeps a scalar string. The field is therefore scalar on most documents and multi-valued on CommandHistory documents. In ES|QL most functions and aggregations return null and emit a warning on multi-valued rows, so queries and rules over process.command_line behave differently for these documents. The events also get process.command_line without the matching process.args / process.args_count that split_command_line produces for every other command line.
This also moves away from the direction already agreed on this PR (populate process.command_line only when the history holds a single command); the previous revision's data-loss problem is better solved by leaving the field unset for multi-command buffers than by making it an array.
Recommendation:
Keep the full list in crowdstrike.CommandHistory and only promote it to process.command_line when it is unambiguous:
if (!remaining.trim().isEmpty()) {
commands.add(remaining.trim());
}
ctx.crowdstrike.CommandHistory = commands;
if (commands.size() == 1) {
ctx.process = ctx.process ?: [:];
ctx.process.command_line = commands.get(0);
}Then regenerate test-fdr-command-history.log-expected.json and test-windows.log-expected.json so the two-command document has no process.command_line and the single-command documents carry a scalar string.
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| description: >- | ||
| Split crowdstrike.CommandHistory on U+00B6 PILCROW SIGN into an array | ||
| and assign the result to both crowdstrike.CommandHistory and | ||
| process.command_line. Placed after split_command_line so that processor |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml:1058
The rewritten description misstates why the processor is safe and drops the note that the pilcrow delimiter is undocumented by CrowdStrike; restore the delimiter rationale and describe the actual guard.
Details
The description now claims the processor is "Placed after split_command_line so that processor only runs on string-valued command lines from crowdstrike.CommandLine." That is not what the ordering does: split_command_line reads process.command_line (populated from crowdstrike.CommandLine), and it is the ctx.crowdstrike?.CommandHistory instanceof String condition, not the position, that keeps this processor off already-converted array values. The sentence is also grammatically broken.
The previous revision recorded that U+00B6 PILCROW SIGN is not formally documented by CrowdStrike but is consistently observed as the delimiter in FDR CommandHistory events. That is the non-obvious reason a magic character appears in the source, and it is the one piece of context a future maintainer cannot recover from the code; the rewrite removed it.
Recommendation:
Restore the delimiter rationale and state the real guard:
- script:
description: >-
Split crowdstrike.CommandHistory into an array of individual commands
and mirror it to process.command_line. CrowdStrike uses U+00B6 PILCROW
SIGN as the delimiter between commands in the history string; this
character has no formal documentation from CrowdStrike but is
consistently observed in FDR CommandHistory events. The
`instanceof String` guard keeps the processor idempotent once the
field has been converted to an array.
tag: script_command_history_to_process_command_line_8c3e1f92🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
| ctx.process = ctx.process ?: [:]; | ||
| ctx.process.command_line = commands; | ||
| on_failure: | ||
| - append: |
There was a problem hiding this comment.
Severity: 🔵 Low confidence: high path: packages/crowdstrike/data_stream/fdr/elasticsearch/ingest_pipeline/default.yml:1085
The on_failure append processors added by this PR have no tag and use an ad-hoc error message instead of the standard ingest.on_failure* template every other handler in this pipeline uses; align them with the existing convention.
Details
Both new script processors (script_command_history_to_process_command_line_8c3e1f92 here, and script_process_interactive_logon_type_9b3d6c7a further down) attach an on_failure append that omits the tag field and writes a custom string carrying only _ingest.on_failure_message.
Every other on_failure handler in this pipeline uses a tagged append with the full template that also records the failing processor type, tag and pipeline, for example append_error_message_4ef54c75 on the json processor. Without those fields an error.message from these handlers cannot be traced back to the processor that produced it, and the untagged processors will be flagged by elastic-package check once the package moves to format_version 3.6.0 or later.
Recommendation:
Use the pipeline's standard tagged handler on both new scripts:
on_failure:
- append:
tag: append_error_message_8c3e1f92
field: error.message
value: 'Processor {{{_ingest.on_failure_processor_type}}} with tag {{{_ingest.on_failure_processor_tag}}} in pipeline {{{_ingest.on_failure_pipeline}}} failed with message: {{{_ingest.on_failure_message}}}'🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
⚠️ Automated review — verify suggestions before applying.
Review summaryIssues found across the latest commits d5f78fe…f5cd816 (12 commits) — 1 medium, 2 low
Issues found across earlier commits c9f2ec7…54efc99 (34 commits) — 1 high, 1 medium, 1 low
Issues found across earlier commits d0257c7 — 1 medium
🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills
|
💚 Build Succeeded
History
|
|
Tick the box to add this pull request to the merge queue (same as
|
Proposed commit message
Checklist
changelog.ymlfile.